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ABSTRACT 


This study analyzes the impact various capabilities have on intelligence gathering 
missions for a Marine Expeditionary Brigade (MEB) commander’s 2015 unmanned aerial 
vehicle (UAV). The Marine Corps Warfighting Lab (MCWL) is developing 
requirements for an intelligence, surveillance, and reconnaissance (ISR) UAV that 
supports rapid planning and decision making for multiple concurrent operations and 
facilitates maneuver and precision engagement. Additionally, acquisition of a 2008 
Pioneer replacement is underway at Marine Corps Systems Command 
(MARCORSYSCOM). The importance of various capabilities for this replacement UAV 
presently lacks quantitative analysis. Through modeling, agent-based simulation, and 
data mining, this study explores the validity of current requirements and provides insights 
into the importance of various UAV characteristics, such as airspeed, endurance, sweep 
width, and sensor capability. The results have design consequences for MCWL’s Fleet 
Battle Experiment Sea Viking 20XX, its largest annual experiment, and provide key 
parameters for physics-based simulations such as COMBAT XXI. The advantage of 
tactical routing, a seven hour (or greater) on station time, a minimum 4,500 meter sweep 
width, and a probability of classification of at least 0.4 are verified for the Sea Viking 
scenario. This analysis indicates that a UAV in this scenario does not need to travel in 


excess of 200 knots. 
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THESIS DISCLAIMER 


The reader is cautioned that the computer programs presented in this research may 
not have been exercised for all cases of interest. While every effort has been made, 
within the time available, to ensure that the programs are free of computational and 
logical errors, they cannot be considered validated. Any application of these programs 


without additional verification is at the risk of the user. 
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EXECUTIVE SUMMARY 


Unmanned Aerial Vehicles (UAVs) are a key component of today’s Intelligence, 
Surveillance and Reconnaissance (ISR) missions. UAVs provide intelligence, a dynamic 
retasking capability, and real-time video imagery. The United States Marine Corps is 
currently reviewing its UAV employment. During Operation Iraqi Freedom (OIF), the 
Marine Expeditionary Brigade (MEB) commander utilized two squadrons of Pioneer 
UAVs. Each squadron provided the capability to fly one UAV for up to six hours with a 
maximum range of over 170 nautical miles. The Pioneer can be relieved on station twice 
with current assets and manpower, achieving a total coverage of 18-hours during surge 
operations with the current force structure in each squadron. The Marine Corps desires to 


improve this capability in an efficient and effective manner. 


The planned Pioneer replacement, the Vertical Unmanned Aerial Vehicle, is in 
the beginning stages of conceptualization and currently has an Initial Capabilities 
Document. It is expected to proceed through the design and acquisition process for fleet 
use in 2008. Marine Corps Systems Command (MARCORSYSCOM) provides guidance 
regarding the relative importance of the capabilities this VUAV could possess, such as 


speed, endurance, and sensor performance. 


The Marine Corps Warfighting Lab (MCWL) has been tasked with answering a 
vague question. Headquarters Marine Corps (HQMC) wants to know “the required MEB 
ISR capability sets needed in order to meet the following 2015 Expeditionary Maneuver 
Warfare requirements? 

1. Support the commander’s rapid planning and decision-making process. 


2 Maintain a comprehensive ISR network to support multiple concurrent 
expeditionary operations. 


3. Facilitate operational maneuver and precision engagement.” 
(Commandant’s Sub Team Guidance, 2003) 
Based on recent performance and a dynamic retasking capability, the UAV has 
been determined to be the key component of the sensor network required to meet these 
requirements. Detailed analysis is required to determine the capability set for this MEB 


commander’s UAV of the future. 


X1X 


There are over 57 UAVs in development or production by corporations in the 
United States and over 216 worldwide. Each has its own unique capabilities and design 
factors. These design factors combine with various uncontrollable factors like 
geography, terrain and enemy capabilities to form a very difficult problem when 
attempting to determine the most important factors and the appropriate needs of the 
Marine Corps. A problem that, even with the fastest computers, most efficient 
simulations, and a team of analysts, would take more than a life time to answer using 
traditional experimental designs. A smarter design is required to comprehensively 


explore how these factors affect a UAV’s ability to perform in expeditionary operations. 


This study looks at UAV operations in the Sea Viking scenario provided by 
MCWL in the MANA agent-based modeling environment utilizing robust design, 
Orthogonal and Nearly-orthogonal Latin hypercubes, data farming techniques, the Maui 
High Performance Computing Center, and the JMP Statistical Discovery Software 
package. The Sea Viking Fleet Battle Experiment, the Marine Corps’ largest annual 
experiment, provides a credible scenario for model development. The model uses 
stochastic techniques to consider the effects of terrain, the enemy portrayed in the Sea 
Viking scenario, and UAV operations from the MEB on the Amphibious Readiness 
Group off the coast of Camp Pendleton. A sample screen shot of a typical starting 


condition is shown below. 
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Six UAV elements are explored: routing, time, number of UAVs, speed, sweep 
width, probability of classification, and employment considerations. Five enemy 
capabilities are also considered: detection range, stealth, time critical target frequency 
and duration, and relationship with neutrals or non-combatants. In all, over 130,000 
mission simulations produce the measures of effectiveness: proportion of enemy 


classified per hour and proportion of enemy classified per mission. 


Classification and Regression Trees (CRTs) provide a way to analyze the 
relationship between factors and the MOE. A regression tree is a recursive partition of 
the raw data into sets of inputs containing similar responses. Partitioning of the data 
occurs successively according to the optimal splitting value determined from all possible 
values of each available variable. The optimal splitting value is the value of the predictor 
variable that minimizes sum of square error among all predictors. After each split, the 
next optimal split is determined within each partition. This may be the same variable as 
the initial split or a different variable obtained from all available factors and can be 
different for each partition. Considering each partition conditionally independently of the 


previous partitions automatically accounts for interactions. 


The CRT on the following page is a recursive partition of the raw data from all 
43,560 MANA runs on all controllable and uncontrollable factors for the one UAV 
scenario. As partitioning of the data proceeds, the most significant factors produce the 
first splits. The partitioning point for a factor range suggests an upper or lower limit for a 
factor capability producing significant improvement in the second MOE: proportion of 
enemy classified per mission. Each box (or node) indicates the optimal factor to partition 
upon and the optimal level of the split itself. Details within the box include the number 
of data points within the node, the mean enemy classification proportion per mission, and 


the standard deviation within the node. 
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MOE 2 Classification and Regression Tree 


(Proportion of Enemy Classified per Mission) 






















¥ “All Rows 
Count 43580 
Mean —0.1861308 
Std Dev 0.0752491 









































































































































y ~ Max Time<7 y ~ Max Time>=7 
Count 24120 Count 22440 
Mean 0.119277 Mean —0.2102282 
StdDev 0.0816309 StdDev 0.058163 
| “ UAVRSW<4134.2 y| “ UAVRSW>=4134.2 y ~ UAVPClass<0.7 y ~UAVPClass>=0.7 
Count $280 Count 15840 Count 15840 Count 6600 
Mean — 0.0512608 Mean 0.1419491 Mean 0.186416 Mean 0.287556 
Std Dev 0.0232316 Std Dev 0.0534919 StdDev 0.049414 StdDev 0.0314 
> Candidates > Candidates 
y “UAVPClass<0.4 |“ UAVPClass>=0.4 | * UAVRSpeed< 156.306 Yy ~UAVRSpeed>=156.306 
Count 3960 Count 11880 Count 3960 Count 11880 
Mean —0.0854043 Mean  0.1607973 Mean  0.1577655 Mean 0.195867 
StdDev 0.0321587 StdDev 0.0452708 StdDev 0.0544517 StdDev 0.0436418 
>| Candidates >| Candidates > Candidates 









































y| * Max Time<3.88 ||| ¥ ~ Max Time>=3.88 


Count 3960 Count 7920 
Mean 0.1157507 Mean, 0.1833206 
Std Dev 0.0223951 StdDev 0.0360758 


> Candidates > Candidates 




















As expected, endurance, referred to as “max time,” is the primary factor when 


considering the total amount of enemy classified during a mission. A UAV on station at 
least seven hours will classify nearly twice the proportion of the enemy than a UAV with 
fewer than seven hours on station time when averaged over all the other variables. 
Additionally, given seven hours to search, a UAV with any sweep width or low 
probability of classification will perform reasonably well. This suggests that endurance 


can make up for moderate short comings in sensor capability. 


A similar analysis on the proportion of enemy classified per hour reveals that the 
most important factor when considering time sensitive Intelligence Preparation of the 
Battlespace in the Sea Viking Scenario is the sweep width of the UAV. In general, wider 
sweep widths yield higher expected proportions of enemy classified each hour—as much 
as twice as much. This is qualified by the assumption that the sensor package can 
maintain a fairly high probability of classification as the sweep width increases. In less 
than seven hours, a UAV/sensor package capable of producing a probability of 


classification of at least 0.4 over a 4,500 meter sweep width may be expected to produce 
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a rate of enemy classification nearly three times greater, on average, than a UAV that 
does not meet these standards for the scenario detailed in this study. This may be crucial 


in a time-constrained situation. 


Whether considering a rate of classification or the proportion classified for an 
entire mission, use of tactical routing is more effective than traditional search patterns. 
This makes tactical sense and lends credibility to the model. For employment 
considerations, when more intelligence is available, it is more important to follow 
preplanned routing as opposed to chasing unclassified contacts. The balance between 
reactivity and strictly following a route is difficult to quantify. With that caveat, 
reserving about one-third of the on station time for chasing unknowns and using the 
remainder to follow a tactical route appears to be the best combination for the Sea Viking 
scenario. A large sweep width and low probability of classification may result in too 
much wasted time if reactivity is high. Conversely, high reactivity can be effective if the 


sweep width is low. 
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I. INTRODUCTION 


A. OVERVIEW 

The United States Military engages in conflicts, peacekeeping operations, and 
power projection around the world. As the world’s greatest superpower, the nation 
expects ever-greater achievements worldwide with less military force committed, fewer 
American casualties, and lower costs—all faster than ever before. While no force may be 
able to stand up to the U.S. in combat, the fog of war often creates uncertainties and 
difficulties resulting in casualties. The military attempts to lift this fog of war and reduce 
uncertainty through Intelligence, Surveillance, and Reconnaissance (ISR) missions. The 
more knowledge of the battle space our commanders have, the greater their ability to plan 


and execute a successful mission with minimal losses. 


Unmanned Aerial Vehicles (UAVs) are a key component of today’s ISR missions. 
UAVs provide intelligence, a dynamic retasking capability, and real-time video imagery. 
The Intelligence Officer for the First Marine Expeditionary Force (I-MEF) indicated this 
combination proved to be invaluable to our forces on the ground during both Operation 
Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF). While the operators 
believe current UAV assets are effectively employed, they are limited. UAVs are in great 
demand and only the highest priority missions receive their support. (Howcroft, 2003) 
As a result of Operation Iraqi Freedom Major Combat Operations, lessons learned 
indicate that a “better asset to collect battlespace intelligence is crucial to the way 
forward.” (US Joint Forces Command, 2004) UAVs provide an alternative to 


complement manned aircraft and satellites in filling this gap. 


The United States Marine Corps is currently reviewing its UAV employment. 
During OIF, the Marine Expeditionary Brigade (MEB) commander utilized two 
squadrons of Pioneer UAVs. Each squadron provided the capability to fly one UAV for 
up to six hours with a maximum range of over 170 nautical miles. The Pioneer can be 


relieved on station twice with current assets and manpower, achieving a total coverage of 


18-hours during surge operations with the current force structure in each squadron. The 
Marine Corps desires to improve this capability in an efficient and effective manner. 


(Hirsch, 2003) 


B. BACKGROUND AND MOTIVATION 

The planned Pioneer replacement, the Vertical Unmanned Aerial Vehicle 
(VUAV), is in the beginning stages of conceptualization and currently has an Initial 
Capabilities Document (ICD). It is expected to proceed through the design and 
acquisition process for fleet use in 2008. (Headquarters Marine Corps, 2003) Marine 
Corps Systems Command (MARCORSYSCOM) provides guidance regarding the 
relative importance of the capabilities this VUAV could possess, such as speed, 
endurance, and sensor performance. If the requirement is for the VUAV to fly at 320 
knots, and one design achieves only 310 knots, should it be ignored? What if the slower 
alternative has a significantly greater endurance or sweep width? What is significant? Is 
it worth the added cost to develop a VUAV capable of 400 knots or is 220 knots 
sufficient, at substantial savings? These alternatives require analysis because intuition 


and experience alone may not provide the best answer. 


The Marine Corps Warfighting Lab (MCWL) has been tasked with answering a 
vague question. Headquarters Marine Corps wants to know “the required MEB ISR 
capability sets needed in order to meet the following 2015 Expeditionary Maneuver 
Warfare (EMW) requirements? 

1. Support the commander’s rapid planning and decision-making process. 

2: Maintain a comprehensive ISR network to support multiple concurrent 

expeditionary operations. 

3. Facilitate operational maneuver and precision engagement.” 

(Commandant’s Sub Team Guidance, 2003) 

Based on recent performance and a dynamic retasking capability, the UAV has 
been determined to be the key component of the sensor network required to meet these 
requirements. Similar to the questions MARCORSYSCOM has regarding the VUAV, 
detailed analysis is required to determine the capability set for this MEB commander’s 


UAV of the future. 


The procedure for determining the future capability requirements for this UAV 
involves demanding maximum performance based on_ expected technology 
advancements. This is not only speculative, but may be overkill for some capabilities. 
For example, the Marine Corps may not need a UAV capable of 72 hours endurance or 
500 knots air speed for the 250 nautical mile max radius expected for MEB expeditionary 
operations. (Hirsch, 2003) There is no formal analysis of the trade space provided by the 
various capability characteristics. Currently, once the requirements have been set, 
meeting them is a pass or fail situation. That is, if a competing contractor proposes a 
UAV, the product either meets all parameters or does not. If a UAV under consideration 
were to fly 2 knots slower than required, it would fail. What if that product had 
endurance 10 times greater than the others? MCWL does not have an analysis tool to 


assist them in determining the value of one capability characteristic over another. 


There are over 57 UAVs in development or production by corporations in the 
United States and over 216 worldwide. The UAVs employed by the US military today 
fly at speeds between 40 and 340 knots, with a combat radius from one nautical mile to 
an over the horizon capability. Some can be launched by hand, some from a ship, and 
some require a full runway. The sensors on board these UAVs have various sweep 
widths, resolutions, sampling rates, and weather effects. (American Institute of 


Aeronautics and Astronautics, 2004) 


These design factors combine with various uncontrollable factors like geography, 
terrain and enemy capabilities to form a very difficult problem. A problem that, even 
with the fastest computers, most efficient simulations, and a team of analysts, would take 
more than a life time to answer using traditional experimental designs. A smarter design 


is required. 


Each year, the Navy/Marine Corps team conducts Fleet Battle Experiment Sea 
Viking in Southern California. The primary objective is Command and Control and ISR 
development. This is an opportunity to validate future combat systems and purposed 
implementation concepts. It also provides a scenario for basing and possibly validating a 


model. (Marine Corps Warfighting Lab website, 2004) The model scenario for this 


study adopts that from Sea Viking 2004, making the results applicable for developing 
employment techniques and capabilities to be evaluated in future Sea Viking 


experiments. 


C; BENEFITS 

This thesis provides benefit in five main areas. First, it yields insight into the 
relative importance of various UAV capabilities in ISR missions for expeditionary 
operations. This directly assists in the development of the 2008 Pioneer replacement 
VUAV currently under acquisition. It aids in determining the required capabilities of a 
system of UAVs to meet the future missions of the MEB commander in 2015. The 
research provides possible mission profiles and employment techniques for validation 
during future Sea Viking Fleet Battle Experiments. The tactics and procedures evolving 
from modeling supports initial Navy doctrine for integrating unmanned vehicles into 
maritime missions specifically addressing current issues from the Fleet. Finally, this 
thesis provides focus for future analysis involving physics-based simulations such as 


Combat XXI and determines key parameters for consideration. 


D. THESIS FLOW 

The following chapter contains a discussion of Agent-Based Models (ABMs) in 
Modeling and Simulation (M&S) and the ABM used for this thesis, Map Aware Non- 
uniform Automata (MANA). The scenario upon which the model is based and the 
representation of entities in the model is also presented. Chapter III examines the 
model’s controllable and uncontrollable factors effecting UAV operations, robust 
experimental design uses for this work, and the implementation of Orthogonal and Nearly 
Orthogonal Latin Hypercubes. Chapter IV discusses the data farming process, post 
processing of the data from batch runs, and data analysis. The final chapter presents 
tactical insights and suggests some possible follow on work for future Operations 


Research theses. 


I. MODEL DEVELOPMENT 


All models are wrong, but some are useful. 


-George Box 


This chapter provides a brief overview of Agent-Based Models (ABMs), 
discusses how their use contrasts with other combat models available today, and 
introduces the ABM Map Aware Non-uniform Automata or MANA. The scenario 
implementation, including terrain and agent portrayal in the MANA environment based 


on Fleet Battle Experiment Sea Viking 2004 is discussed. 


A. AGENT-BASED MODELING 

In today’s world of high-priced, high-tech systems with competing alternatives 
and joint considerations, decision makers require detailed program analysis. Often expert 
opinion and historical references do not provide adequate information for this analysis. 
The systems may even be too new to have “experts.” Live Test and Evaluation (T&E), 
while part of any system development, is often too costly to explore the full range of 
possibilities that warrant consideration. Furthermore, T&E tends to be one of the first 
areas cut when time or money is tight. (Hoivik, 2003) This is where modeling and 
simulation can provide performance expectations and insights. The Defense Modeling 
and Simulation Office (DMSO) goal is to: 


Provide readily available, operationally valid environments for use by 
DoD components: 


- To train jointly, develop doctrine and tactics, formulate operational 
plans, and assess warfighting situations. 


- To support technology assessment, system upgrade, prototype and full 
scale development, and force structuring. 


Furthermore, common use of these environments will promote a closer 
interaction between the operations and acquisition communities in 
carrying out their respective responsibilities. To allow maximum utility 


and flexibility, these modeling and simulation environments will be 
constructed from affordable, reusable components interoperating through 
an open systems architecture. 


(Defense Modeling and Simulation Office website, 2004) 


Significant problems with most currently used combat models include time and 
manpower requirements. Building a data base, implementing a scenario, completing a 
statistically sufficient number of runs, processing the output, and conducting an analysis 
often requires months. Teams of specialists develop these combat models. Often the 
team members have a thorough understanding of only a small portion of the model that is 
finally produced. Tying all the pieces together can prove to be the most difficult 
developmental piece. The Joint Simulation System (JSIMS) was conceived as the 
Department of Defense’s (DoD’s) “Flagship” simulation to model joint combat 
operations. Despite over a billion dollars spent on the development of this system, 
difficulties in integrating the many programs used in the system may be the end of the 


initiative. Many in the field believe this program is already dead. (Manago, 2004) 


It is not uncommon for the databases developed for a combat model to have 
questionable accuracy. Sometimes databases may hold unclassified parameter values for 
developmental or training purposes. Discovery and correction of errors in the data entry 
process, whether unintentional mistakes or undocumented temporary guesses, does not 
always occur during verification. The 1999 DMSO award winning simulation Joint 
Conflict And Tactical Simulation (JCATS) is one of the key training models employed 
today. During a training exercise it was discovered that the sensor for a Tomahawk Land 
Attack Missile (TLAM), a long range, subsonic cruise missile, was in fact represented by 


an 8x magnification binocular. This is a rather serious error. (Manago, 2004) 


However, this is not an attempt to question current M&S efforts. These examples 
merely bring to light the point of the quote at the beginning of this chapter by George 
Box: “All models are wrong, but some are useful.” In the end, all the physics equations 
and detailed parameters of high resolution, physics-based models feed into a combined 
probability or weight. This probability or weight feeds into another complex equation 


designed to determine a hit, detection, movement, or other outcome. 


Another approach to modeling and simulation is from the ground up. Rather than 
attempting to simulate activity as close to reality as possible by modeling every detail, 
why not model only those entity attributes which have a significant impact on the 
situation? ABMs, also called “distillations,” follow this philosophy. (Marine Corps 
Warfighting Lab website, 2004) They are distillations of the real world. Individual 
entities, called “agents,” are given capabilities and behaviors. Capabilities may include 
parameters such as movement speed, available weapon systems and sensor attributes. 
Behaviors include factors like an agent’s propensity to follow orders, congregate with 
friendly agents or attack enemies. Each agent is autonomous and reacts according to its 
behavior characteristics and what it perceives within its own situational awareness 
picture. The interactions of the agents on the digital “battlefield” result in scenarios 


which resemble key characteristics of the real world in which we live. 


Most ABMs are stochastic with each iteration of a scenario resulting in a 
different outcome. (Lucas, 2003) Execution of many iterations produces expected 
outcomes and identifies variations. Uncommon results, or outliers, can be the most 
interesting cases. Combining this with the ABM’s easy set up and modification 
characteristics, quick run time, and high performance computers allows for exploration of 
a wide range of parameter values. This results in a powerful tool for an operations 


analyst’s toolbox. 


Data farming is a technique commonly used in conjunction with ABMs. The 
purpose of data farming is to explore the effects of a parameter in a model across its 
range of possible values. Changing a parameter and viewing its effects through multiple 
iterations of a simulation enables effective estimation of the impact that parameter has. 
Completing this process for all parameters of interest ascertains the significant 


parameters. (Brandstein and Horne, 1998) 


Project Albert is a division of the MCWL which utilizes data farming and ABMs 


asa 


method to address decision-maker's questions that applies high performance 
computing to modeling in order to examine and understand the landscape of 


potential simulated outcomes, enhance intuition, find surprises and outliers, and 
identify potential options. (Marine Corps Warfighting Lab website, 2004) 


One of the many ABMs in the Project Albert suite is Map Aware Non-Uniform 


Automata. 


B. THE AGENT-BASED MODEL MANA 

Map Aware Non-uniform Automata, or MANA, is the agent-based modeling 
environment selected for this research. David Galligan and Michael Lauren began 
development of MANA for the New Zealand Army and Defense Force in 1999. Contact 
information is provided on the opening MANA screen (Figure 1) providing tribute to the 
work. Since then, MANA has been added to the data farming tools in the Project Albert 
suite of ABMs maintained by MCWL. 
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Figure 1. Map Aware Non-uniform Automata opening screen provides contact 
information. 


In the MANA environment, the agents are: 


>» Map Aware—Agent’s situational awareness includes the depicted terrain as 
well as battlespace activities in the simulation. 

> Non-uniform—Each agent may have different behavior parameters and 
capabilities. That is, they do not all have to move or act the same way. 

>» Automata—Agents react independently on the battlefield according to their 
own individual situational awareness and personalities. 


This distillation allows for graphical depiction of the terrain and agents to the 
desired level of detail. MANA is a straight forward application that is intuitive and easy 
to use with a well developed Graphical User Interface (GUI). The data farming 
techniques built in provide the ability to explore an extensive range of capabilities in 
minimal time. More details are readily available in the MANA User’s Manual. 


(Galligan, 2003) 


Developing the terrain for a scenario is a fairly easy process making use of the 
graphical editor built into MANA which produces basic scenario maps. Ifa scenario map 
is available, it can be converted digitally with any graphical editing software, such as MS 
Paint, Paint Shop Pro, or even Power Point, into a format MANA recognizes. In this 
ABM, terrain characteristics affect an agent’s movement speed, cover, concealment, and 
line of sight. These settings are relative to each other and/or conversions from tactical 


parameters. 


Agent parameters in MANA are in four basic types: personality weightings, move 
constraints, basic capabilities, and movement characteristics. The personality weightings 
determine an agent’s propensity to move toward or away from battlefield entities such as 


waypoints, cover, concealment, and other agents. 


Move constraints are conditional modifiers to the personality weights. For 
example, a UAV agent may be more likely to seek out and follow enemy agents when it 
is within a certain distance of the objective. MANA enables use of real world 
performance parameters for basic capabilities for conversion to the scale of the 
simulation scenario. Basic capabilities include parameters, set in the real world, that are 
converted to the scale of the scenario. These include maximum speed, sensor range and 
ability, and weapon range and effect, among others. Finally, movement characteristic 


parameters determine the type of algorithm used and the degree of randomness employed 
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by an agent. This affects behaviors like obstacle avoidance and the effects of terrain. 
Over 200 parameters may be employed in 49 possible states for each agent, providing the 
ability to capture a broad range of behaviors and capabilities of battlefield entities. 


(Galligan, 2003) 


The personality page from the Edit Squad Properties menu, displayed in Figure 2, 
shows how an agent’s propensities for movement may be modified from -100 to 100 and 
given an effective range. In this case, the agent’s propensity to move toward the next 
waypoint is 20, indicating a desire to move toward it. A negative propensity value 
indicates a propensity to move away, such as this example shows for the top category, 
enemies. Additionally, MANA allows for determining the effective range of a 
personality weighting. In this case, if a way point is between 0 and 1000 cells away, the 
indicated weighting applies. This agent also has its greatest propensity to move toward 
cover. The behavior for this agent may be summarized as desiring to stay covered or 
protected while moving towards its next waypoint and staying away from enemy 
contacts. The ratios created from these weightings feed the movement equation along 
with other parameters to determine the agent’s most desirable move. A detailed 
description of how the movement propensities affect the agent’s movement can be found 


in MANA’s users guide. (Galligan, 2003) 
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Figure 2. Example of MANA Edit Squad Properties page, tab one of nine: 
Personality. The display shows some of the many attributes which affect an agent’s 
behavior on the battlefield in the MANA environment. 


The support for ABMs in the Project Albert suite is excellent. It ranges from 
analysis to modeling, and provides the capability to make large batch runs remotely via 
Internet. Users can farm hundreds of thousands of data points from their own worksite. 
Scenario development support for MANA users is available from the software developers 
themselves. It is not uncommon for the programmers to provide version updates within a 
month of emerging requirements to capture a key element or vary a parameter not 


available in the current version at a user’s request. 


During the time frame of this study alone, many updates and versions to the 
MANA program have been released. This study uses only version 3.0.29 because of its 
methodology for computing line of sight (LOS). Subsequent versions calculate (LOS) in 
an undesirable manner for this study. If a line between observer and target crosses an 


obstruction square (i.e., with 0.92 concealment in MANA) then line of sight is not 
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possible. To determine LOS, version 3.0.29 calculates the probability of seeing through 
each square between the observer and a target in range. Newer versions compute this 


based on relative elevations. 


Figure 3 is a visual representation of the difference between the two methods. On 
the left is the LOS depiction for an agent in “Dense Brush,” “Light Brush, and “Billiard 
Table,” or an open area, for the version used in this study. On the right is the LOS 
depiction for the same agent in the same environments for subsequent versions. The 
table in the middle displays the values for Going, Cover and Concealment for various 
terrain types. Notice the various levels of Concealment offered by the Billiard Table, 
0.00 in black, Light Brush, 0.30 in light green, and Dense Brush, 0.90 in dark green. 
Intuitively, an agent should see less in Dense Brush than in Light Brush or the Billiard 


Table terrain. This characteristic is only demonstrated in version 3.0.29. 


© fii Terrain Propertion 











Figure 3. Line of sight determination in version for this study versus subsequent 
versions. We see the desirable LOS characteristics of version 3.0.29 on the left, 
versus subsequent versions on the right. [Best viewed in color] 


C. SEA VIKING SCENARIO 

The Sea Viking Fleet Battle Experiment, developed by MCWL, provides the 
scenario for the model to explore important UAV capability questions. Sea Viking is an 
experimentation program designed to allow exploration of ways in which the MAGTF 
can be transformed to increase combat power, operational versatility, utility, and 


deployability. It is a Navy/Marine Corps exercise for developing C4ISR techniques and 


12 


tactics that improve our ability to focus forward presence and Operational Maneuver 
From The Sea (OMFTS). The desired end-state is to make the Ship-To-Objective 
Maneuver (STOM) concept an operational reality for the Navy/Marine Corps Team. 
(Marine Corps Warfighting Lab Website, 2004) 


The annual experiments involve imposing a reality-based threat scenario from a 
current area of interest on operations in the Southern California area. Utilizing Naval 
Base San Diego, Marine Corps Air Station (MCAS) Miramar, Marine Corps Base (MCB) 
Camp Pendleton, March Air Reserve Base, MCAS Yuma, and MCB 29 Palms, the full 
range of tactical operations may be conducted with Marines and sailors from the ship to 


the objective via air, land, and sea. 


The scenario from Sea Viking 2004 provides a validated threat and mission 
context for the model which is acceptable to the principal stakeholders in UAV 
development for the Marine Corps. Mountain Infantry units, Coastal Infantry units, an 
Armor unit, and Time Critical Targets (TCT) are included in the force structure. The 
host nation is expected to provide no support. The geographic region is approximately 
150 nautical miles by 150 nautical miles. Details from the Sea Viking Scenario are in 
Appendix A. The focus for this study is on the Intelligence Preparation of the 
Battlespace (IPB) during the initial phases of operations for the MEB. This directly 
addresses the first portion of the EMW requirements posed to MCWL above, as well as 
many of the questions from the VUAV Project Manager. (Marine Corps Warfighting 
Lab, 2003) 


D. INSTANTIATING A SEA VIKING SCENARIO IN MANA 

This section presents the modeling concepts and includes some examples where 
appropriate to provide an understanding of the level of detail in this study. This includes 
a description of the battlespace, agent development, aggregation, and creative 
alternatives. More information on MANA functionality and algorithms may be found in 
the MANA User’s Manual. The details of the model development and the final model 


are available by contacting the author or advisors. 
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The model includes representations of the Southern California terrain, Sea Viking 
enemy organization and capabilities, as well as civilian (neutral) presence. Friendly force 
representation is limited to UAV operations in support of the ISR mission. Figure 4 
displays a screen shot from the start of a typical run. The numbered circles identify the 


different friendly and enemy units modeled. 


For orientation purposes, the display shows Los Angeles and San Diego city areas 
in dark yellow and 29 Palms (the objective) in the upper right corner. Bright yellow 
depicts all major and most secondary roads. Over 515 agents make up the scenario, 
including 345 civilians, 9 enemy tanks, 150 enemy infantry, 10 enemy TCTs, and 1 to 3 
friendly UAVs. 


Area | includes the UAVs, two in this case, at sea on Naval ships. Each iteration 
of the scenario starts with a different random number seed which varies the initial 
position of all agents on the battlefield. The UAVs have a planned tactical route to 
follow over land to cover key tactical objectives. Area 2 encompasses the Red (enemy) 
Coastal Infantry agents from the Sea Viking scenario intended to patrol the beach line. 
Area 3 identifies the Red Low Infantry which operate in low level terrain outside the city. 
The mission for this group is to move towards the city in search of neutrals to convert to 
enemy. The Red Mountain Infantry are in area 4. The mountain infantry move toward 
the choke point at the juncture of Banning Pass and Yucca Valley. Area 5 to the 
Northeast includes the Red Objective Area Forces. Infantry on the objective are 
protecting the senior leadership. Some of their forces move down Yucca Valley to meet 
up with the forces coming down from the mountains. In the Northwest is the Red Armor 
Unit within area 6. The armor moves southeast to take a defensive position in the 
vicinity of Banning Pass. Center stage in area 7 are the TCTs located in the channelized 
terrain representing a Surface-to- Air Missile (SAM) systems or Anti-Aircraft Artillery 


(AAA). Throughout the region are yellow “Neutral” or civilian agents. 
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Figure 4. MANA Sea Viking base scenario. The display shows a sample initial 
friendly, enemy and civilian agent layout within the battle space. [Best viewed in 
color] 


Key enemy characteristics include movement speed, dispersion, tendency to seek 
cover & concealment, sensor range, enemy-—neutral cohesion, stealth or detectability 
characteristics, and a mission. UAV parameters include speed, endurance, sweep width, 


sensor capability, reactivity and number of units employed. 


1. The Battlespace 

As mentioned, terrain depiction in MANA is straight forward. Multiple JOGAIR 
maps of the Southern California Area make up the official Sea Viking 2004 Scenario 
map. (Marine Corps Warfighting Lab, 2003) The area of operations is identified, 
captured digitally, and enhanced, tracing over roads, urban terrain, water, desert, and light 
and dense vegetation areas in visually appropriate colors, with specified RGB values. 
After being converted to bitmap format, the image is imported to MANA. Using the 
MANA Scenario Map Editor, the values for the parameters that determine how these 
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different terrain features affect an agent’s speed, concealment, and cover are entered. 
Many of the values used are default settings in MANA. Others result from previous 
MANA work, interpolation, and the tactical experience of several Marine officers. 


Figure 5 depicts the values from the MANA Scenario Map Editor. 
(Oo x! 


Scenario Map 
Editor 


t Edit Terrain Properties | Select Map- 





Terrain 





a ca a 
oo 0 0 


BilliardT able §1.00 


| © Background 





192 
64 Brush Width: |10 — 


255 


wo 


elect Terrain 


255 (© BilliardT able 
180 Wall 
| 


& 
© Hilltop 
200 © Road | Key [Value 
102 © LightBush | Going 1.00 
196 © DenseBush = Cover 0.00 
Conceal 0.00 








Restore 


Dera 


Terrains 








Load Picture 


eee) 
Save Picture 


(993, 620) 








Figure 5. Terrain effects parameter values as displayed in the MANA Scenario Map 
Editor. 


Terrain parameter values for Road, Light Brush, and Dense Brush utilize the 
default settings, make intuitive sense, and have been utilized previously in other studies. 
(Brown, 2000) For example, the terrain type “Road” has a parameter value of 1.00 for 
“Going,” meaning an agent’s speed parameter value is multiplied by a factor of 1.00, or 
unaffected, since roads are not intended to limit movement. Conversely, the “Cover” and 
“Conceal” parameters for the “Road” terrain type are both zero since roads provide no 
cover or concealment. Terrain types that provide cover have a value greater than zero but 


less than one, indicating the probabilistic effect the terrain has on the likelihood that an 
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agent in that terrain would be seen by another agent or hit if shot at. Concealment 
providing terrain only affects an agent’s probability of being seen. The parameter values 
for City, Desert, and Water terrain types were developed for this study. They also make 
intuitive sense and were reviewed with sample scenarios in the MANA environment to 


ensure an effective representation. 


MANA displays maps from bitmap files in a maximum resolution of 1000 by 
1000 cells. This setting results in each cell equating to about 295 meters on a side for the 
area of operations represented. Each time step in MANA equates to 36 seconds in the 
real world. A spread sheet provides an effective means to compute, display, and 
reference many desired conversions. Figure 6 displays details for this scenario using 
Microsoft Excel for computing the bounding corners of the scenario in latitude and 
longitude, conversions to nautical miles, statute miles, kilometers, meters, feet and 
MANA cells. This information is used in developing movement and sensor capabilities 


for the agents in the MANA model. 
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Due to the size of the geographic area in the scenario, consideration is given to the 
variation in distance between lines of longitude. At the latitude for the center of the Sea 
Viking Fleet Battle Experiment, one degree of longitude is equivalent to about 49.83 
nautical miles vice 60 nautical miles at the equator. Figure 6 shows the effect of this 


detail in the MANA battlefield. 


2. Agent Development 

The values for the four types of agent parameters, mentioned previously, are 
developed by a variety of means. Some parameters, such as agent movement speed, can 
be easily determined based on known real world values converted to the scale of the 
model. For example, a tank which travels at a maximum speed of 60 miles per hour in 
the real world will move 1.09 cells per time step in this MANA scenario. Others are 
more difficult to determine, such as a UAV’s propensity to move towards the next way 
point. Difficulties arise when attempting to determine the value to use for that same 
UAV’s propensity to move toward, or follow, detected enemy. What should be the ratio 


between these two competing goals? 


This is where data farming comes in. Farming is the act of running the simulation 
for multiple iterations at a variety of levels to determine the effect of that parameter on 
the scenario outcome. These results provide appropriate values to set for questionable 
parameters to produce a behavior that makes logical and tactical sense. Variations of 
significant factors in the final analysis help to focus on the effects of these factors and 


their interactions with other variables in the model. 


The primary focus of this study is the ability of a UAV to detect and classify 
forces on the ground. To this end, priority is on modeling UAV sensor capabilities as 
well as its endurance, speed, routing, and tactical employment. Any particular sensor has 
a specified Field of View (FOV). The sweep width of the UAV/sensor combination 
changes with altitude. For example, a 15° FOV sensor employed at 2000 feet Above 
Ground Level (AGL) has a sweep width of about 527 feet. This same sensor employed at 
14,000 feet AGL will have a sweep width of about 3686 feet. The predominate factor in 
this situation is sweep width. With this in mind, sweep width is modeled for this study 
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with the knowledge that a FOV/flight altitude combination may be derived for a given 


sweep width. 


Figure 7 permits determination of the sweep width for a given combination of 
FOV and flight altitude. The plots show how quickly a 10,000 meter FOV is achieved at 
a moderate altitude considering that potential design altitudes for UAVs are as high as 
40,000 feet AGL. The vertical axis provides the sweep width in meters. The horizontal 
axis lists altitudes up to 25,000 feet AGL. Each line represents a different FOV. Thus, 
for a particular FOV, the left axis indicates the sweep width obtained at a given altitude. 
Sweep width is varied from 2,000 meters to 10,000 meters in this study to capture 


significant possibilities. 
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Figure 7. Graph of sweep width obtained given a flight altitude and FOV. The 
display shows the range of possible sweep width values. 


The other sensor factor which deserves attention is probability of 
detection/classification. For a given sensor, the single glimpse probability of 


detecting/classifying a given target at a given range is generally known—under ideal 
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conditions. Ideal conditions may include weather, solar/lunar position, atmospheric 
conditions, clutter, contrast, and signal strength. For example, an optical sensor my have 
a probability of detection of 0.7 for a tank from 10,000 feet AGL with clear skies, a 
relative humidity of less than 20%, and the sun overhead. As cloud cover increases, 
humidity increases, the sun changes position, a dust storm pops up, or condensation 
builds up on the lens, this probability will drop. Also, higher altitudes are associated with 
reduced resolution, signal strength, increased interference as well as other confounding 


effects on probability of detection/classification. 


In this MANA model, single glimpse probability is the probability of classifying a 
target in one time step. As the sensor sweeps an area, a target in the area may pass down 
the center of the sensor path or may be near the edge. If the target is near the edge, it 
may only be possible to acquire it on a single time step. In this case, target classification 
likelihood is the single glimpse probability. On the other hand, if a target passes down 
the center of the sensor’s path, it may be in range for several time steps. In this case, the 
probability of classifying this target is additive, using the laws of probability and 


assuming independent glimpses, for each time step it is within the sweep width. 


Figure 8 displays a sampling of the various resulting lateral range curves modeled 
in this study. The maximum and minimum capability UAVs, in regard to resulting 
probability of classification, are presented along with three intermediate examples. Each 
line provides the probability of classifying a target for the given distance from the UAV’s 
flight path indicated along the x-axis. The most capable sensor nearly equates to a cookie 
cutter with a probability of detection of 1 within about 9,500 meters. This UAV is 
traveling at the slowest setting, 100 knots, with the widest sweep width, 10,000 meters, 
and the highest probability of classification, 0.9, for a single glimpse. 
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Figure 8. A visual sample of the lateral range curves modeled. The display shows 


the resulting probability of classification as a function of lateral range from the UAV 
for a sensor with various probabilities of classification at various airspeeds. 


In this study, the actual single glimpse probability assigned to the sensor varies, 
accounting for possibilities beyond that which exists or is expected in the near future. 
This demonstrates another advantage of the data farming process: the ability to determine 
if a significant advantage may result from a capability outside of what is currently under 
consideration. Probability of detection ranges are varied from 0.1 to 0.9, allowing for 
consideration of more highly capable sensors than are currently available as well as 


degradation to current sensor capabilities, due to weather, altitude, or other factors. 


3. Aggregation 

Force aggregation is a common technique in modeling and simulation. The 
primary model used for this study does not have an agent for each soldier described in the 
Sea Viking scenario. Approximately one-third the number of infantry, tanks, and TCTs 
are portrayed in the primary (aggregated) model to reduce the run time of the MANA 
software package. The aggregated model runs in less than thirty seconds where as the 
full model can take over four and a half minutes on a 2.81 GHz Pentium 4 with 512 MB 
of RAM. This makes a big difference in the time required to complete hundreds of 


thousands of runs. The robust design implementation (Section 3.3) to explore the desired 
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variety of factors would take over a year of CPU time to complete with the full 
implementation of the enemy forces in the Sea Viking scenario. A consideration is 


whether this representation of only one-third the forces is valid. 


A validation test with the full level of enemy forces run many times at high, 
medium, and low levels for all variables can determine if this aggregation scales 
properly. These levels are similarly run in the aggregated model. The resulting data 
provides the proportion of enemy classified in both the full model and the aggregated 
model under equivalent conditions at three different levels. Analysis of this data using 
the nonparametric Kruskal-Wallis Test (Conover, 1999) indicates that the distribution of 


detection ratios produced from each of these models are statistically indistinguishable. 


Figure 9 displays plots of two of the levels. The vertical axis provides the 
proportion of enemy classified for a particular run, which is listed on the horizontal axis. 
The graph shows the similarity between the non-aggregated and aggregated models at 
high levels, roughly between 0.2 and 0.3, and the low levels, below 0.05. The conclusion 
from the aggregated and full comparison is that analysis based on the aggregated scenario 


is valid for the full scenario in regard to classification proportion. 
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aggregated scenarios displaying the equivalence of the resulting distributions for high 
and low factor levels. [Best viewed in color] 
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4. Measure of Effectiveness and Creative Modeling Alternatives 

MANA enables access to a variety of parameters, including over 200 available for 
data farming, and Measures of Effectiveness (MOEs) which can be evaluated for each 
simulation run. The primary MOE for this study is proportion of enemy detected over 
time. MANA can record detection data for each agent at each time step. While this 
provides high resolution on the MOE, for such a large scenario, the output files 
eventually crash the operating system due to a limit on the number of subdirectories 
within a directory. Utilizing the default output from MANA provides a more efficient 
sampling of the MOE that proves sufficient. The default output provides a summary 
statistic, number of agents killed, for each type of agent following each scenario run. 
Over multiple iterations at each set of parameter values, or design points, a mean and 


standard deviation for the number killed for each agent type is obtained. 


This study focuses on ISR, looking at UAV classifications, not at its ability to 
destroy the enemy. This is where the creative modeling comes in. By providing the 
UAV with a weapon having a probability of hit of 1.0 and unlimited rounds, the UAV 
“kills” each agent it detects and classifies. Since the UAV flight path in this scenario 
does not cover the same terrain more than once, multiple detections, or lack thereof, is a 


minimal concern. 


False alarms occur when a UAV attempts to classify an unknown contact and it 
turns out to be a neutral vice an enemy. The MANA output contains this information in 
the form of number of neutrals classified. The expected outcome is a ratio between 
neutral classified to enemy classified. Since we have a two to one ratio of neutral to 
enemy agents that appear on the battlefield this ratio would be about two to one on 


average. 
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Hil. DESIGN OF EXPERIMENTS 


A challenge in conducting analysis on various UAV capabilities is the large 
number of factors, their wide range of levels, and their interactions. This chapter presents 
the factors in two sets, controllable and uncontrollable. Discussion follows regarding the 
utilization of methods to effectively explore the parameter space, robust design and 
orthogonal Latin hypercubes (OLH). The intent is to relay the value and capabilities of 
each technique as well as provide an understanding of their employment. References 


provide a more thorough understanding of how to utilize robust designs or OLHs. 


A. CONTROLLABLE FACTORS—UAV CAPABILITIES 

The controllable factors potentially affecting a UAV’s ability to detect enemy 
units and vehicles on the ground not only have a large amount of variability, but also 
have significant interactions. It may seem obvious that the desired case is a wide field of 
view (FOV) and high resolution yielding a high probability of detection. In the real 
world, as FOV increases, resolution decreases for a particular sensor. Similarly, it may 
appear desirable to have the fastest UAV possible to cover the most area. However, there 
are two confounding factors in this case. First, the faster the UAV travels, the less 
endurance it generally has. This may result in a requirement for more UAVs to maintain 
coverage of an area. Second, the faster a UAV searches an area the less time is spent on 


any one location, driving detection probabilities down. 


Each of these factors should be considered in system design to develop a UAV to 
accomplish a desired mission. This study focuses on the UAV’s ability to detect enemy 
forces during the intelligence preparation of the battlefield in a MEB-sized operation. 
The controllable UAV factors determined to be of greatest importance through 
preliminary analysis and discussions with the sponsor include: 


= Routing 

=" Number of UAVs employed 

" Time available 

= Speed 

= Sweep width (function of Altitude and FOV) 
* Classification probability 

= Reactivity 
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The effect of the first of these factors, UAV routing, was explored in some preliminary 
work conducted during the Project Albert International Workshop 8 (PAIW 8) in 


Singapore. The remaining factors are the focal point of this modeling and analysis effort. 


The route a UAV follows on any given search mission affects the number of 
enemy forces detected and classified. If the UAV does not fly over any enemy locations, 
there are no detections or classifications. In military operations today, forces often have 
some idea of where the enemy is likely to be, through satellite imagery, ground 
intelligence, or terrain analysis of avenues of approach. We do not rely on random or 
generic search pattern techniques for this type of employment. Tactical routing, designed 


around where our forces are going and where enemy forces are likely to be, is preferred. 


Preliminary analysis examined various routing considerations. At PAIW 8, a 
team of defense analysts from a variety of countries developed several independent routes 
for the Sea Viking scenario. Some developed routes having no knowledge of the enemy 
locations, while others possessed knowledge of general enemy locations, such as, “An 
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enemy armor unit is expected to be located north of the large urban area.” Each route 
required starting and ending at the ship off the coast. Of the sixteen different routes 
produced, three categories emerged. One category was a lawn mower type search 
pattern, another uses knowledge of likely enemy locations to set the UAV waypoints, and 


the last uses tactical routing. 


The study produced four final routes, three from the categories described above 
and one previously approved as likely routing by MCWL. As an example, the routing 
approved by MCWL appears in Figure 10. It was originally developed based on terrain 
analysis and location of the objective area. While there would be different waypoints for 
any UAV commander who planned this mission, this one is as likely as any that might be 
planned. It includes the urban areas and routes most likely to be traversed on ingress, 


channeling terrain, key defense points, and the objective area itself. 
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Figure 10. An example of routing for a single UAV approved by MCWL. 


The focus on routing as one of the primary variables yielded 10,560 MANA runs 
across 264 design points (combinations of input factors). Figure 11 shows the distribution 
of the outcomes from the traditional search pattern type routing and the tactical routing. 
The results clearly indicated that tactical routing was superior to random or lawnmower 
type search patterns. The 95% confidence intervals for the two subsets do not overlap, 
and the tactical routing is over 60% better, on average, for this scenario. This makes 
intuitive sense and provides support of the model. Based on these outcomes, the current 


study uses the tactical routing approved by MCWL for all runs. 
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Figure 11. Distributions of the proportion of enemy classified form Traditional 
Search Patterns versus Tactical Routing of the UAV in the SV scenario. Note the 
tactical routing proves to be 63% better on average. 


The number of UAVs employed affects how much area can be covered and how 
long it takes to complete the search. Interactions with speed and sweep width are 
intuitive, but must be quantified. Are two UAVs flying at 150 knots better than one 
flying at 300 knots? How do different sweep widths affect the performance? An upper 
limit of three UAVs has been selected for this study based on discussions with the 
sponsors and the Sea Viking scenario. It has been determined to be unlikely that the 
Marine Corps will have the funding or personnel to employ more than three UAVs 
simultaneously as standard operating procedure for a Marine Expeditionary Brigade 


(MEB) size Area of Responsibility (AOR). (Hirsch, 2003) 


Time and space separate multiple UAVs active in the scenario. This follows 


general tactical airspace control rules which indicate that aircraft commanders should 
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plan for deconfliction by a minimum of two of the following three means: time, space, or 
altitude. (Department of Transportation, 2004) This has the added benefit of ensuring 
there are no multiple detections possible in the scenario. All of these points support the 
modeling and data collection method described in the previous chapter, using kills to 


record classifications. 


Time in the AOR, run time, in conjunction with the proportion of the enemy 
classified provides a measurement of the expected rate of proportion of enemy classified 
over time for a UAV with a particular set of capabilities. In the real world, a commander 
may send out a UAV to search an area, but the information coming back to the command 
center is constantly monitored and, at a minimum, hourly updates are reported. In the 
simulation, each iteration stops at a predetermined time or when the UAV arrives back at 
the ship. Collecting the statistics on number and type of enemy detected up to the stop 
point provides a single data point for that set of capabilities. Many iterations of the 
scenario are run for each set of capabilities, yielding sample means to enable examination 


of the relative effectiveness of a set of capabilities over time. 


Endurance is a difficult factor to model. Developing different routing for each 
endurance level is not practical for this study. Instead, estimating the effects of time 
using the methods described above provide insight into endurance. Routes are varied 
from one enabling a single slow UAV to complete a route to a set of three routes for three 


fast UAVs. 


It should be noted that many other factors affect the required endurance level for a 
MEB sized operation. The first is the many possible variations in scenario parameters 
regarding distance from launch site to target area. Second, for Intelligence Preparation of 
the Battlespace (IPB), knowing how much information can be expected hour by hour may 
be just as important as mission by mission. Finally, these UAVs are currently intended to 
relieve each other on station, moving the significance of endurance considerations from 
the focus of this study to deck cycles and launch and recovery issues. These tactical 


considerations certainly warrant attention. Manned aircraft generally have priority on the 
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ship and the physics of how well a UAV fits into the deck cycle for shipboard operations 
may be a larger driving factor. For this study, the endurance needed to cover the SV area 


is assumed. 


In concert with the Sea Viking 2004 scenario, the UAV starts each mission from a 
ship just off the coast. Limiting run time to halt each iteration at a specific experimental 
point or back at the ship yields the desired performance measure, number of enemy 


classified, up to that point. 


UAV speed is a representation of the airspeed the UAV flies at for the mission. 
There is no variability in airspeed during a single iteration of the scenario. Airspeed in 
the MANA environment is obtained by multiplying the desired real world airspeed by the 
nautical mile conversion factor, 6.2713, which converts a nautical mile to a MANA cell 
width for the geographic area represented. This number is then divided by three to keep 
all possible airspeed values within the parameter limits of MANA. This scaling by a 
factor of 1/3 is conducted for all movement speeds in the scenario, thereby keeping all 


speed relations equivalent. 


The range of speeds considered is from current UAV capabilities to a speed 
comparable with some manned aircraft. The low end is 100 knots and the upper end is 
400 knots. There are many proponents for a UAV which will have dash speed over an 
Osprey, which has a maximum airspeed of 305 knots (Global Securities, 2004). While 
dash speed is not specifically considered in this scenario, the impact of this airspeed 


capability is considered for completeness. 


UAV sweep width describes how wide the search area is for a single glimpse by 
the sensor. Sweep width is modeled using the sensor range parameter in MANA. 
Specifically, the UAV agent’s detection and classification ranges are varied in lock step. 
A UAV agent detects a target as a function of the UAV’s detection range, line of sight, 
the target’s stealth, and available concealment on the terrain. Once detected, a target is 
then classified based on the UAV’s probability of classification parameter, discussed 


below. 


Sweep width ranges from 2,000 meters to 10,000 meters to encompass the current 


UAV capabilities and future possibilities. Employing a UAV at over 25,000 feet AGL is 
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a possibility for the MEB commander and Figure 7 in the previous chapter displays the 
possible sweep widths obtained for various FOV. Although there are several UAVs on 


the market today which operate at altitudes over 40,000 feet AGL, the sweep width range 
in this study encompasses current expectations for the type and size UAV under 


consideration by the Marine Corps. 


UAV classification probability is the probability that a detected target is classified 
as friendly, enemy, or neutral on a single glimpse. In this model, the single glimpse 
probability of classification is held constant throughout the range of the sensor for any 
single iteration. The resulting effect as the UAV/sensor combination travels over the 
ground is that agents on the edge of the sensor range have a lower probability of 
classification than targets which the UAV passes directly over and are within range for a 
longer period of time. This provides a different lateral range curve for each sweep 
width/probability combination. The probability of classification range in this study is 
from 0.1 to 0.9, effectively capturing the full range of possibilities. 


UAV reactivity is a term to describe the UAV’s propensity to follow unclassified 
agents or to stay on its route. It is how “reactive” the UAV is to contacts. The intent is to 
explore the possible employment considerations a UAV commander may have. Given 
any area to search, the UAV commander will ideally plan a route to cover the entire area. 
The question is, once flying the route, should the UAV commander send the UAV after 
unclassified contacts in an attempt to determine if they are enemy, or should he follow 


the planned routing to ensure complete coverage? 


In MANA, UAV reactivity is modeled by changing the UAV’s propensity to 
follow detected “unknown” or unclassified agents relative to its propensity to move 
toward the next way point. This ratio is varied from one-third as desirous to move 
towards unknowns to three times as desirous to move towards unknowns in MANA 
personality weights. In any one time step, the UAV agent will consider this ratio along 
with the proximity of other UAVs and contacts already classified as enemy in 


determining its next move. 


Figure 12 presents the basic set up for a UAV in MANA. The key factors, 


movement speed, classification range, and classification probability are set at a middle 
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level for the base case and varied in the experimental design. These middle factors 
permit a realistic run time viewing that is easy to understand and follow visually for 


debugging purposes. 
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Figure 12. Sample UAV Range tab from Edit Squad Properties function in MANA. 
The display shows basic speed, sensor detection range, and sensor classification range 
and probability functional areas. 


B. ROBUST DESIGN AND NOISE FACTORS 

Robust design was pioneered by Genichi Taguchi in the 1980s for quality 
planning and engineering product design activities. (Taguchi and Wu, 1980) The fact 
that often a process may contain many variables which may be uncontrollable, or costly 
to control, can weigh heavily on the best course of action to take in attempts to optimize 


that process. For example, if the enemy has the ability to detect an approaching UAV 
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before becoming within range of the UAV’s sensors, the enemy may hide and perhaps 
avoid detection. If the enemy’s detection range were known, a UAV could be 
constructed which has a sensor range greater than that of the enemy. However, the 
enemy’s detection range is variable. In the case of counter detection, it can depend on 
wind speed, wind direction, flight altitude, and background noise. Additionally, the cost 
of increasing the sensor range of a UAV sensor is generally a loss in resolution of that 
sensor, monetary costs aside. (Federation of American Scientists, 2004) This loss in 


resolution may be too great a cost for effective classification of targets. 


Traditional experimental designs attempt to hold uncontrollable variables, or 
‘noise’ variables, constant. This is intended to ensure that the impact due to these 
uncontrollable variables is constant throughout the experimental runs. This results in an 
“apples to apples” comparison and some mean performance indicator or Measure of 
Effectiveness (MOE) may be obtained. However, this also results in decisions made on a 
narrow set of circumstances. A robust design ensures the controllable factor levels are 
optimized with regard to the uncontrollable variables that affect performance. (Sanchez, 
1994) By exploring the influence of noise variables, a set of design parameters, which 
may not perform the best in a particular instance, may perform best across a wide variety 


of circumstances in which the US military finds itself employed these days. 


The intent of distillations is not to model every aspect of reality, but to focus on 
the significant factors and relationships. (Brandstein and Horne, 1998) The significant 
factors can be effectively determined through iterative evaluation of the noise variables. 
The end result is a subset of the significant factors which are uncontrollable in the real 


world but whose effects can be explored through M&S. 


c, UNCONTROLLABLE FACTORS—ENEMY CAPABILITIES 

In developing this model, several factors, not typically under the control of the 
UAV developer or MEB commander, require investigation. The following factors were 
determined to be significant enough to warrant inclusion in the subset of uncontrollable 
factors in the final analysis: 


= Enemy detection range 
= Enemy stealth 
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= Neutral-Enemy cohesion 
*" TCT vulnerability frequency 
* TCT vulnerability duration 


Enemy detection range is fairly self-explanatory and covered in a previous 
example. Enemy stealth is equally intuitive. It represents how well a target proceeds 
unnoticed. In the MANA environment, this is the weighted probability that an agent is 
not seen by a particular agent in any one time step regardless of other concealment, 
cover, or line of sight factors. These two factors are used in conjunction with a MANA 
state change. When the enemy detects the approaching UAV, it conducts a state change 
to an enemy contact state where the agent possesses a higher stealth to simulate a duck 


and cover reaction. 


Enemy-Neutral cohesion represents the ability of combatants to congregate with 
neutrals in an attempt to avoid detection. The MANA parameter controlling this aspect 
of the agent’s behavior is the propensity to move towards neutrals. Along with the 
agent’s desire to seek concealment, easily traversed terrain, and their next waypoint while 


maintaining dispersion, this is a weighted value in the movement equation. 


Time Critical Target vulnerability frequency is a measure of how often TCTs 
leave their hide site to move or engage targets, for example. Similarly, TCT vulnerability 
duration is how long they leave their hide site for these activities. These factors are 
controlled with the various agent state changes that are available in the MANA set up. In 
its hide site, a TCT has a high stealth value ensuring that the probability of detection is 


extremely small. When the TCT is out of its hide site, it is vulnerable to detection. 


A base set up for a typical enemy infantry squad is displayed in Figure 13. The 
agent is given a negative weight for “uninjured friends” to provide the appropriate 
dispersion. Positive weights for other factors ensure the agent moves toward its intended 
objective considering concealment, without working too hard, and integrating with 


neutrals. 
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Figure 13. Red Mountain Infantry squad basic personality configuration. Values are 
manipulated to control dispersion, provide a mission, effect movement propensities, 
and manipulate cohesion with neutrals. 


D. ORTHOGONAL LATIN HYPERCUBES 


There are a large number of factors worthy of consideration between the 
controllable and uncontrollable factors. A problem arises in attempting to effectively 
vary these factors across a wide range of possible levels. A traditional factorial 
experimental design tests only a few factors at two or perhaps three levels each. To 
utilize this approach, some factors would have to be left out of the experimental design 


and linear relationships assumed. A smarter design is required. 


An Orthogonal Latin hypercube (OLH) design is chosen for its excellent space 
filling properties, the resulting low correlation between factor inputs, and ability to 
identify nonlinear relationships. (Cioppa, 2002) OLHs_ can be used to design an 
experiment evaluating up to seven factors at 17 levels each in an efficient and effective 


manner. Nearly orthogonal Latin hypercubes (NOLHs) have nearly the same properties 
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with slightly higher, but negligible, correlation between factors. NOLHs can be utilized 


to evaluate from 8 to 22 factors at up to 129 levels. 


Figure 14 plots each design point derived from an eight factor NOLH with 33 
levels for each factor. This NOLH is then crossed with a two-factor factorial design. The 
factor names are presented down the diagonal. Each point on the plot represents the 
corresponding factor levels for a design point. The NOLH’s space filling properties are 
demonstrated by the plots in contrast to the factorial design factors. Notice the lack of 
space filling represented by the factorial factors, #UAVs and Routing, compared to any 
of the other factors whose design points were obtained from a NOLH implementation. 
This design allows for the exploration of many variables (in this preliminary case, ten) 
over a large range while evaluating many points within the range for each appropriate 


variable. This ensures the ability to identify nonlinear relationships and interactions. 
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Figure 14. Pairwise scatter plot of design points utilizing a nearly orthogonal Latin 
hypercube crossed with a factorial design. Factor names appear along the diagonal. 
Each dot represents a design point for the corresponding factors. 
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The orthogonal nature of the design results in no significant design-imposed 
correlation. This provides the ability to look at the effects of each variable independently 
as well as interactions during analysis. Through optimal chaining of OLHs or NOLHs, 
the space filling characteristics can be increased while maintaining the orthogonal nature 
of the design and no significant design point correlations. (Cioppa, 2002) This provides 


a greater ability to analyze multidimensional data. 


This study uses two, optimally appended, orthogonal Latin hypercubes for each 
group of factors, controlled and uncontrolled. This provides 33 design points for each 
group optimized for greatest space filling. The two sets of 33 design points each are then 
crossed to ensure each of the controllable factor design points are evaluated at the 
maximal range of possible combinations of uncontrollable factors as the robust design 
section discusses above. This results in 33 times 33 = 1089 design points which are each 
run for the three factorial cases of one, two, or three UAVs. Finally, these 3267 design 
points are each run for 40 iterations to take advantage of the stochastic nature of MANA 
and provide a look into the variation that may be expected. This results in 130,680 total 
MANA runs of the scenario. Conducting this many runs with a typical, physics-based 
combat model, such as JCATS or Combined Arms and Support Task Force Evaluation 
Model (CASTFOREM) would take years. This advanced level of experimental design 


with typical combat models is undesirable due to the resources required. 


The ranges for the controllable and uncontrollable factors in both the real world 
and the MANA environment are displayed in Figure 15. On the right is a sample of the 
orthogonal Latin hypercube for the controllable factors presenting the first 20 design 
points. In addition to the 40 replications run for this design, a separate set of runs of 5 


repetitions each is available to be used as a test set for verifying the models. 
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Figure 15. Design of Experiments for the primary analysis. The display shows the 
factors, factor ranges, required number of runs, and a sample of the design points and 
levels for the controllable factors. 


This experimental design took just under 48 hours to run at the Maui High 


Performance Computing Center (MHPCC). Using traditional full factorial designs, this 


analysis would require over 1.8 x 10'S runs. This would not finish running on the fastest 


computers available today before the sun burns out. A graphical comparison is provided 


in Figure 16. 
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Figure 16. Graphical comparison of the number of runs required to conduct this 
analysis with a traditional full factorial design versus a smarter design with 
orthogonal Latin hypercubes. [Best viewed in color] 
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IV. DATA ANALYSIS 


Orthogonal Latin Hypercubes (OLHs)/Nearly-Orthogonal Latin Hypercubes 
(NOLHs) and MANA output facilitate post processing and data analysis. A quick review 
of the collection and preparation of data for analysis begins this chapter. Next is a 
discussion of the statistical modeling techniques utilized to gain an understanding of the 
relationship between the Measure of Effectiveness (MOE) and the predictor variables and 


the results they produce. 


A. DATA COLLECTION AND POST PROCESSING 

MITRE Corporation in Woodbridge, VA coordinated over 150,000 total 
production runs for this analysis. MANA iterations run on site produced the preliminary 
data. The second data set was completed during Project Albert International Workshop 8 
(PAIW 8) in Singapore. The final set of runs was completed utilizing the assets at the 
Maui High Performance Computing Center (MHPCC). MITRE facilitated the execution 


of each experimental design ensuring proper implementation at the appropriate site. 


Each experiment returned a comma delimited file easily converted into an Excel 
spreadsheet or JMP statistical discovery software format. Both programs, from Microsoft 
and the SAS institute respectively, are commonly used for data manipulation and 
analysis. (JMP User’s Manual, 2002) The output from each run includes the variable 
levels, duration of the run, and the number of killed agents, classified by type. The 
number recorded in each run is equivalent to the number classified, as previously 
described in Chapter II. Spreadsheet calculations easily turn the MANA units back into 


real world values for effective analysis in a user friendly format. 


The MOE is expected enemy classification proportion per hour. This value is 
computed for each MANA run from the output as follows. First, the number of enemy 
classified in that run is divided by the total number of enemy in the scenario. This is 
done to provide the proportion classified for all 130,680 runs. The proportion classified 
is divided by the actual run time for each run to yield the proportion classified per hour. 


The last computation is average classification proportion per hour across the 40 runs at all 
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3267 design points. This number provides the average proportion of enemy classified per 
hour for the given factor levels in that design point—including controllable and 


uncontrollable factors. 


The statistics of the aggregated data over all controllable factors for the one UAV 
scenario are displayed in Figure 17. The average proportion of enemy classified per hour 


over all runs is 0.0293. 
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Figure 17. Distribution of the MOE, mean proportion of enemy classified per hour, 
for the one UAV scenario. Notice the mean is 2.93% of the enemy classified per 


hour. 


Using 40 iterations at each design point ensures that the distribution function for 
the random variable (mean proportion detected per hour) representing the MOE for a 
particular data point is approximately normal, as stated in the Central Limit Theorem. 
This ensures the data meets one of the general assumptions for regression analysis. 
Randomly selected design points evaluated using the Shapiro-Wilk Test for Normality 
verify normality of the response and its use as our estimator of the true effects of the 


particular parameter settings. (Conover, 1999) 


Figure 18 is a Normal Quantile Plot of the data for a randomly selected design 
point. Data that is normally distributed tends to fall on the diagonal line. Data contained 
within the confidence interval bounds is said to be distributed normally with 95% 


confidence. (JMP User’s Manual, 2000) Notice that this data falls almost entirely on the 
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line and is all contained within the confidence intervals. Thus, at this typical design 


point, the MOE fits well to a normal distribution. 
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Figure 18. Normal Quantile Plot of resulting enemy classification proportion per hour 
from a randomly selected design point, number 2678. Most of the data fall on the 
diagonal and all fall within the 95% confidence interval indicating the normality of 
the measure of effectiveness for a typical design point. 


After data post-processing, the resulting spreadsheet includes a row for each 
design point listing the factor levels and the associated mean enemy classification 
proportion per hour. The data are now ready for multiple regression and decision tree 


analysis. 


B. MULTIPLE REGRESSION ANALYSIS 

Multiple regression is a common technique for determining the effect of various 
factors on a response variable. It involves applying linear combinations of the 
coefficients of the factors that predict the response variable by minimizing error. 


Minimizing the error term produces an accurate fit of the response based on the factors. 
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Various statistical packages are available for facilitating multiple regression analysis. 
JMP Statistical Discovery Software version 5.0.1a (JMP User’s Manual, 2002) is utilized 


for this work. 


A detailed description of the data analysis process follows for the one UAV 
scenario and is similar for each model. The details of the models developed for the two 
and three UAV scenarios can be found in Appendix B. Due to anomalies with the multi- 
UAV runs requiring further iterations and time constraints on this study, further analysis 


is recommended for the multi-UAV scenarios. 


The single UAV model utilizes 1089 responses to regress the controllable and 
uncontrollable variables on the MOE, mean classification proportion per hour. This first 
considers all main effects, two-way interaction terms, and main effects quadratic terms 
resulting in a total of 76 candidate terms for consideration in the model. Stepwise 
regression pairs down the parameter space to only those factors with a specified 
significance level by incrementally adding and deleting terms to the regression model. 
Once the statistically significant factors are identified, to obtain a parsimonious model, 
additional factors may be removed if they provide minimal improvement to the fit. (JMP 


User’s Manual, 2000) 


During the development of the model, performing stepwise regression on the 
factors eliminates those factors below the 0.01 level of significance. The resulting model 
provides parameter estimates for the significant factors and an R-squared value for the 
entire model. The R-squared value is the proportion of the variation in the MOE 
explained by the model. An R-squared value of 1.0 means that the model perfectly fits 
the data. At this point, an iterative process of removing the term with the least 
significance, reconstructing the model, and evaluating the resulting fit of the model 
provides a means for selecting a preferred model. The preferred model provides a 


balance between simplicity and goodness of fit. 


Figure 19 demonstrates this process for the single UAV scenario. The R-squared 
value, or proportion of explained variation in the MOE, is plotted as a function of the 
terms indicated along the X-axis. The line with diamonds represents the values obtained 


from a regression of controllable and uncontrollable factors, interactions, and squared 
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terms utilizing 1089 design points each of which is the mean of 40 points. The line with 
squares represents the R-squared values obtained from a regression on just the 


controllable factors using 33 aggregated design points. 
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Figure 19. Graph of fit of one UAV models by term. The figure shows the similarity 
between the models containing both controllable and uncontrollable factors and the 
models containing only controllable factors. Additionally, the preferred model with 

eight terms is indicated at the point of diminishing return in the fit on the graph. Note 
that all terms retained are controllable and there are no interactions between 
controllable and uncontrollable factors. [Best viewed in color] 


Notice the similarity between the two plots and how closely they follow each 
other. This is a visual representation of the lack of practical significance of the 
uncontrollable variables in this scenario over the ranges explored. Analysis of each of 
the other two models, for two and three UAVs produces similar results. The preferred 
model, even with the uncontrollable factors considered, still contains only controllable 
factors. This leads to a simpler model focusing on just the controllable variables and 


allows for the aggregation of the data from 1089 points to 33 points without significant 
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loss of information. The remaining analysis considers the mean enemy classification 
proportion per hour for the 40 times 33 = 1320 runs at each controllable factor design 


point. 


Figure 19 also displays the distinct point of diminishing returns as factors are 
added to the model. This typical characteristic of regression models is useful in 
determining the difference between statistical significance and practical significance. 
While all the terms listed at the bottom of the graph are statistically significance at the 
0.01 level, they provide minimal additional explanation of the response after the first 
seven or eight terms. The first eight terms listed along the bottom of the graph explain 
over 90% of the variability in mean enemy classification proportion per hour across all 


the levels explored in this scenario. 


Another way of looking at the effects of each term is to add them into the model 
in reverse order. Figure 20 shows the incremental predictive power each term on the 
horizontal X-axis contributes to the model in terms of R-squared in the response variable 
on the vertical Y-axis. The model from all the factors is the line with diamonds and the 


model obtained from the aggregated data for the controllable factors is the line with 
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Figure 20. Proportion of variation in one UAV mean enemy classification proportion 


per hour across all scenario factor levels as each term is added to the model. There is 
a clear point of diminishing return and similarity between the models with all factors 
and the models aggregated over the uncontrollable factors. [Best viewed in color] 
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Again, the resemblance between the two plots clearly depicts the similarity 
between the two models. Using the simpler model developed from controllable factors 
and aggregated data produces no significant loss of information. In addition, this graph 
more clearly depicts the point of diminishing return. The value of adding the next most 
significant term after the eighth term is minimal. The arrow indicates the desired point of 
balance between goodness of fit and simplicity. The preferred eight term model contains 


the following terms for the single UAV scenario in order of importance: 


UAV Sweep Width 

UAV Probability of Classification 

UAV Sweep Width Squared 

Maximum Time Available 

Reactivity 

Interaction between UAV Sweep Width and Reactivity 
Maximum Time Available Squared 

UAV Speed 
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Note that all these terms are controllable factors. 

UAV sweep width is the most significant predictor of the variability in the MOE 
across UAV capabilities in this scenario. Nearly 40% of the variance in the proportion of 
enemy classified per hour is explained by the UAV’s sweep width alone. The first four 
terms explain over 80% of the variance in the MOE. UAV sweep width, probability of 
classification, sweep width squared, and maximum time available provide a good 


model—explaining the vast majority of the variability all by themselves. 


There is a noticeable dip above the reactivity term in Figure 20, due to the fact 
that, on its own, reactivity does not explain a significant portion of the variability. 
However, the next term, an interaction between reactivity and the UAV’s sweep width, is 
a significant predictor of the variability in the MOE. It is traditional to include the main 
effect term in any model where an interaction term is significant even if the main effect 


term is not. (Whittaker, 2003) Thus, reactivity is retained in this model even though it is 
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not mathematically required. The squared effects of maximum time available and UAV 


speed also have practical significance, although not as great as the previous terms. 


Figure 21 is a visual representation of the preferred model. This plot of the 
predicted versus actual response displays how closely the model explains the MOE. 
Notice how well the data points follow the diagonal—indicating a nicely fit model. The 
residual plot on the right displays the homoscedasticity, or constant error variance, lack 
of influential cases, normal residual distribution, and linear relationship. (Hamilton, 
1992) The associated statistics for this model are provided in Figure 22. The actual 
coefficients for each term in the model are listed. Also of note is the way in which the 
interaction terms and square terms are evaluated. The mean for each is subtracted off for 


each factor included in an interaction or squared term to ensure proper scaling. 
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Figure 21. Predicted versus actual mean enemy classification proportion per hour 
displaying the good fit of the preferred single UAV model with eight terms and 
associated residual plot verifying the absence of pattern in the residuals. 
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Summary of Fit 


RSquare 0.928752 
RSquare Adj 0.905003 
Root Mean Square Error 0.003944 
Mean of Response 0.029291 
Observations (or Sum V¥gts) 33 
Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 
Model 8 0.00486708 0.000608 39.1067 
Error 24 0.00037337 0.000016 Prob=F 
C. Total 32 0.00524045 <,0001 


Parameter Estimates 


Term Estimate Std Error tRatio Prob=}t] 
Intercept 0.0047896 0.003806 1.26 0.2203 
UAVRSpeed 0.0000198 0.000007 2.69 0.0128 
UAVRSVY 0.0000031 2.764e-7 11.27 <.0001 
UAYPClass 0.024338? 0.002762 8.81 =.0001 
UAYUnkAttr -0.000008 0.000028 -0.27 O 7876 
MaxSteps -0.001322 0.000221 5.98 <.0001 
(UAYRSYV-6056 1 2)*(UA YUnkAttr-50) -3.652e-8 1138e-8 -3.21 0.0038 


(UAYRSVV-6056.1 2)*(UAVRSVY-6056.1 2) -7.03e-10 1.36e-10 -515 <.0001 
(MaxSteps-7.00242)*(MaxSteps-7.00242) 0.0002559 0.000084 3.05 0.0056 











Figure 22. Preferred one UAV model. The R-squared value for this model is above 
90%. The display shows the coefficients for each term and the significance of each of 
the terms as well as the overall model. 


This model makes sense. As UAV speed, sweep width, and sensor capability 
increase, the proportion of enemy classified per hour increases. These effects are 
intuitive, as capabilities increase, so does performance, lending credibility to the model. 
Conversely, as the time the UAV spends in the area increases, the proportion of enemy 
classified per hour decreases, requiring slightly more contemplation but also making 
sense. The longer the UAV searches, new contacts become sparse. Additionally, the 
target rich environments are visited first, providing more opportunity for classifications. 
This also makes tactical sense since it is quite reasonable to send a search asset to survey 
the most likely enemy sites first. The rate at which classifications occur decreases over 


time. 


The interaction term indicates the significant relationship between sweep width 
and reactivity. It may seem better to have a wide sweep width. However, when the 


sweep width is high and the UAV is employed with a propensity to follow unidentified 
AT 


contacts until classified, the greater sweep width can accumulate too many unknown 
contacts. This has a negative effect on the proportion classified per hour due to a conflict 
of interests. The UAV cannot follow all unidentified contacts in this case and performs 


poorly. 


Figure 23 displays this interaction in the top-middle and left-center row plots. For 
two different levels of reactivity, 10 and 90 for example, varying the sweep width has a 
different effect. This is depicted in the left center plot by the non-parallel curves. The 
red curved line, with a value of 10, indicates minimal desire to chase unclassified 
contacts and a desire to stay on the route. In this case, as sweep width is increased, the 
point of diminishing return is reached later than in the case depicted by the blue line. The 
blue line, with a value of 90, indicates a high propensity to follow unclassified contacts 
vice follow routing. When the UAV is more inclined to follow the unclassified contacts, 
a point of diminishing return is reached more quickly because it develops too many 


competing interests more quickly. 
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Figure 23. Interaction plots between UAV sweep width, UAV probability of 
classification, and max steps. The display shows the presence of an interaction 
between SW and PClass and the nonlinear effects of SW and time. [Best viewed in 
color] 
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In the top middle plot, the effect of varying reactivity is different for a sweep 
width of 2,067 meters, in red, compared to a sweep width of 10,040 meters, in blue. The 
difference in slopes between the two lines is clear. Increasing the reactivity of the UAV 
with the large sweep width has a negative effect. The wider the sweep width becomes, 
the more unclassified contacts the UAV detects, and the harder it is to classify each. The 
UAV is overtasked. Conversely, when the sweep width is low, increasing the reactivity 


helps the under tasked UAV find more enemy contacts. 


The sweep-width-squared term indicates a point of diminishing returns for sweep 
width. As sweep width increases, holding all other factors constant, the mean enemy 
proportion classified per hour increases up to a certain point. This makes intuitive sense. 
As discussed in the previous chapter, increasing the sweep width without increasing the 
capability of the sensor to achieve the equivalent single glimpse probability of 


classification may result in a decrease in performance at some point. 


The squared term for time the UAV is on station, indicates that there is not just a 
simple linear relationship explained by the main effect term, but that endurance has a 
point of diminishing return as well. Figure 23 displays the interactions of endurance as 
dashed lines indicating no significant interaction with the other variables. Each plot runs 
parallel with respect to the interaction term. However, the nonlinear effect of “max 
steps” can clearly be seen. In both plots on the right side of Figure 23, as max steps 
varies, the mean classification proportion per hour decreases in a curvilinear fashion. 
Although the mean proportion of enemy classified per hour decreases over time, the rate 
of decrease diminishes as well. It appears that about seven hours is the point at which the 


curve levels off. 


The significance of each of these factors can be visualized using leverage plots. 
The leverage plot shows for each point what the residual would be with and without that 
effect in the model. (JMP User’s Manual, 2002) Figure 24 displays the effect each term 
has on mean proportion of enemy classified per hour. As the factor level is varied across 
the bottom, while holding all other terms constant, the effect on the MOE can be 
referenced on the left. The terms with a greater effect have a line with a steeper slope. 


Sweep width has the largest impact and appears with the greatest slope. Probability of 
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classification, the second most significant factor, is very close in significance, as the 
associated plot in Figure 24 is nearly as steep as the sweep width plot. In this manner, 


relative significance can be visualized for each of the effects. 
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Figure 24. Leverage plots of one UAV preferred model terms indicating degree to 


which each term affects the MOE, mean proportion of enemy classified per hour. 


C. CLASSIFICATION AND REGRESSION TREES 
Classification and Regression Trees (CRTs) are good tools for creating decision 
trees and provide another way to analyze the relationship between factors and the MOE. 


A regression tree is a recursive partition of the raw data into sets of inputs containing 
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similar responses. Partitioning of the data occurs successively according to the optimal 
splitting value determined from all possible values of each available variable. The 
optimal splitting value is the value of the predictor variable that minimizes sum of square 
error amongst all predictors. After each split, the next optimal split is determined within 
each partition. This may be the same variable as the initial split or a different variable 
obtained from all available factors and can be different for each partition. Considering 
each partition independently of the previous splits automatically accounts for 
interactions. This continues until the improvement in fit falls below user specified levels. 
Again, we must balance fit with parsimony. The concept is complex, but the resulting 


model is easy to understand. 


Figure 25 displays a recursive split of the raw data from all 43,560 MANA runs 
on all controllable factors and uncontrollable factors for the one UAV scenario. As 
partitioning of the data proceeds, the most significant factors produce the first categories. 
The splitting point for a factor range suggests an upper or lower limit for that factor 
capability producing significant improvement in the MOE. Each box indicates the 
optimal factor and the optimal level to divide upon. Details within the box include the 
number of data points within the split, the mean enemy classification proportion per hour 


and the standard deviation within the split. 


This analysis complements the regression analysis in the previous section. Again, 
all of the terms in the tree are controllable factors. The first split is made on UAV sweep 
width. This is the single most significant factor in this scenario. The decision tree also 
provides the optimal split point at about 4,430 meters. Based on the analysis for this Sea 
Viking scenario, over the ranges examined, a system with a sweep width greater than 
4,430 meters can be expected to provide a rate of enemy proportion classified over two 
times greater, on average, than a system with sweep width below 4,430 meters, 0.0348 


and 0.0166 respectively. 
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Figure 25. Decision tree split on the raw data by proportion of enemy classified per 
hour for each MANA run of the one UAV scenario, considering controllable and 
uncontrollable factors. The tree indicates the overall significance of sweep width and 
probability of classification, and the interaction with reactivity. 


The next most important factor is probability of classification. If sweep width is 
above the optimal minimum of 4,430 meters, ensuring the sensor capability provides at 
least a 40% probability of classification for the expected environmental and geographic 
conditions will provide over a 60% increase in the expected proportion of enemy 


classified per hour. 


The next significant factor explaining performance is reactivity when the UAV 
sweep width is less than 4,430 meters. In this case, employing a more reactive UAV will 
increase the proportion of enemy classified per hour. The additional information gained 
from following unclassified contacts may be acted upon and provides an increase in 


performance. 


Two factors are noticeably missing from the CRT: UAV Speed and Time. These 
factors appeared in the regression analysis, but even when this CRT extends several more 
levels, these factors do not show up as might be expected. Their significance is 


substantially less than the primary factors brought out by the decision tree. 
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This brings up the possibility of another MOE, namely proportion of enemy 
classified per mission. This MOE captures the effect of the entire mission whereas the 
original MOE captures the time effect as a rate. By considering the total enemy classified 


during a mission, the effect of time may become more apparent. 


With this new MOE, or MOE 2, proportion of enemy classified per mission, a 
new CRT tree is developed and appears in Figure 26 below. As expected, time, in this 
case max time, is the primary factor when considering the total amount of enemy 
classified during a mission. A UAV on station at least seven hours will classify nearly 
twice the proportion of the enemy than a UAV with fewer than seven hours on station 
time when averaged over all the other variables. Additionally, given seven hours to 
search, much greater probability of classification is required to produce a significant 
increase in classifications. This suggests endurance may make up for short comings in 


sensor capability. 
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Figure 26. CRT split on the raw data by proportion of enemy classified per mission 
for each MANA run of the one UAV scenario, considering controllable and 
uncontrollable factors. The tree indicates the significant time effect and the 

appearance of speed while retaining previous decision factors. 
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The effect of sensor capability, namely sweep width and probability of 
classification, appear similarly in this model as they did in the previous CRT. A brief 
study of the second decision tree in Figure 26 reveals that, on average, a UAV with a 
sweep width greater than 4,200 meters and a probability of classification of 0.4 or more 
can classify approximately 18% of the enemy in this scenario in about four hours. This 


provides a good reference point in determining factor effects. 


Speed appears in Figure 26 when the UAV is going for over seven hours and has 
a probability of classification less than 0.7. Compared to the other splits, the effect is 
minimal making it the sixth, and last, one to appear. At speeds over 160 knots the 
proportion of enemy classified increases by less than 30% on average. A sweep width 
greater than about 4,200 meters with fewer than seven hours available nearly triples the 


expected proportion of enemy classified. 
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v; CONCLUSIONS 


The purpose of computing is insight, not numbers. 


-Richard Hamming 


A. ANALYSIS SUMMARY 

The data obtained from the design of experiments and the Sea Viking scenario 
implemented in MANA for this study provide normally distributed data points from 
which to conduct statistical analysis. The uncontrollable factors, or noise, imposed in this 
scenario produce statistically significant effects on the MOE, but not practical effects. 
This allows for simplification of the data into aggregated points over the noise and further 


analysis which focuses on the controllable variables. 


This study uses two analysis techniques to look at the proportion of enemy 
classified per hour: multiple regression analysis and Classification and Regression trees. 
The two analyses complement each other. Each analysis identifies similar factors of 
greatest importance, key interactions, and provides similar insights. In contrast, the 
regression analysis yields formulae for predicting UAV performance for capability 
combinations not explicitly modeled in the simulation. Additionally, relative effects of 
one capability set can be compared to others using the information from regression 
analysis. The Classification and Regression Tree analysis provides a hierarchical view of 
the factors. The splits define factor levels as a minimum or maximum goal to keep in 


mind. 


The regression analysis on this aggregate data produces a good fitting model for 
the single UAV scenario. Sweep width and probability of classification have 
dramatically more significant effects on the proportion of enemy classified per hour than 
the other factors for the ranges and factors considered. This makes sense and lends 
credibility to the modeling. Interactions between the factors indicate the degree to which 
the factors must be considered together. As expected, sweep width, probability of 
classification or sensor capability, and reactivity or employment philosophy are related 


and insight is provided regarding these relations. 
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Decision points for primary factors are indicated. A sweep width above about 
4,500 meters provides a significant increase in the proportion of enemy classified per 
hour. A sensor package providing at least 0.4 probability of classification per glimpse for 
contacts within its intended environment will significantly improve performance with 


regard to the proportion of enemy classified per hour. 


The absence of time and speed in the CRT led to the value of a second 
performance measure: proportion of enemy classified per mission. The decision tree 
based on this MOE brings to light the value of having a UAV with an on station time of 
at least seven hours. The significance of speed is once again revealed as minimal in this 


scenario. 


Using the analytical model obtained from this analysis, predictions of the relative 
performance expected for other capability sets in this scenario can be evaluated. The 
decision points provide a threshold value to keep in mind in the development of systems 
with regard to relative expected performance in this scenario. By decomposing the data 
according to the decision points, further analysis can provide a more accurate picture of 


the effects of the factors on the MOE. 


Minor anomalies are present in the two and three UAV scenario which warrant 
additional analysis and iterations. Models based on the current information are in 


Appendix B. They are consistent with the insights for the single UAV scenario. 


B. KEY TACTICAL INSIGHTS 

The most important factor when considering time sensitive Intelligence 
Preparation of the Battlespace in the Sea Viking Scenario is the sweep width of the UAV. 
In general, wider sweep widths yield higher expected proportions of enemy classified 
each hour—often more than twice as much. This is qualified by the assumption that the 
sensor package can maintain a fairly high probability of classification as the sweep width 
increases. In less than seven hours, a UAV/sensor package capable of producing a 
probability of classification of at least 0.4 over a 4,500 meter sweep width may be 


expected to produce rate of enemy classification nearly three times greater, on average, 
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than a UAV that does not meet these standards for the scenario detailed in this study. 


This may be crucial in a time constrained situation. 


In consideration of mission success, or classifying the largest proportion of the 
enemy during the course of a mission, a UAV on station at least seven hours is most 
valuable in this scenario. This time on station may mitigate many of the shortcomings in 
sensor capability or speed of the UAV. On the other hand, with fewer than seven hours 
time on station, the factors discussed above for time sensitive gathering of intelligence 
provide the greatest proportion of enemy classified per mission in the Sea Viking 


scenario. 


Whether considering a rate of classification or the proportion classified for an 
entire mission, use of tactical routing is more effective than traditional search patterns. 
This makes tactical sense and lends credibility to the model. For employment 
considerations, the more intelligence that is available the more important it is to follow 
routing as opposed to chasing unclassified contacts. The balance between reactivity and 
strictly following a route is difficult to quantify. With that caveat, reserving about one- 
third of the on station time for chasing unknowns and using the remainder to follow a 
tactical route appears to be the best combination for the Sea Viking scenario. A large 
sweep width and low probability of classification may result in too much wasted time if 


reactivity is high. Conversely, high reactivity can be effective if the sweep width is low. 


C. ADDITIONAL INSIGHTS 

Many insights have been realized in the course of this analysis. The most 
significant appear in the previous section. The following is a list of additional insights 
surfacing during this work. 


e Speed has a positive effect on UAV classification performance. However, 
speeds greater than 200 knots provide little improvement in ability to 
classify enemy. Certainly there are other important reasons to have a fast 
UAV which should be considered, for example dash speed. 


e Increasing sweep width when minimal enemy intelligence is available will 
increase the proportion of enemy classified. However, there is a point of 
diminishing return when task saturation becomes an issue. This is 
especially true if the probability of classification is too low. 
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Two UAVs do not provide twice the classification ability; however there 
is improvement over a single UAV. Three UAVs seems to have a more 
synergistic effect, doubling the expected proportion classified with two 
UAVs in this scenario given the routes examined. 


Agent-based models and data farming techniques provide an efficient 
means to view the effects of a variety of parameters. Unknown values 
may be farmed to provide insights to parameter effects without explicit 
modeling of capabilities which may be unknown and previously guessed. 


Creative modeling is required in Agent-based Modeling. The point is 
quick turnaround and insights into interactions and focusing further 
analysis. Using readily available MOEs in a particular ABM can enable 
more effective and capable analysis. 


D. FOLLOW ON WORK 


The following is a list of follow on research of value that could be accomplished 


utilizing this work: 


Analysis of factors effecting classification of time critical targets 
Analysis of effects of the terrain 
Data analysis on Multi-UAV data 


Further development and enhancement of the Excel tool for parameter 
exploration 


Focused analysis over the key parameters and ranges identified 
Analysis of the effect of a much larger neutral to enemy ratio 


Repeat analysis in an OIF based scenario 


The following is a list of follow on research of value stemming from this research: 


Validation/comparison of the MANA Sea Viking scenario with a similarly 
developed scenario in Combat XXI 


Development of a model addressing the dynamic retasking issue 


Development of a model addressing the multiple concurrent operations 
issue 


Development of a model focusing on the Global Hawk Maritime 
Demonstration for the Maritime Patrol and Reconnaissance Force 


Analysis of the available sensor packages and development of expected 
probabilities of classification 
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Reliability study determining the number of UAVs required to support a 
Sea Viking scenario 


Human factors study determining effective screening and classification 
techniques integrating the man and the machine 


Analysis of distributed communications flow for a network centric 
environment incorporating multiple UAVs for sensing and shooting 
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APPENDIX A. SEA VIKING 04 SCENARIO DETAILS 


The following slides are taken directly from the Sea Viking Scenario brief and 
describe the enemy situation for the scenario. (Marine Corps Warfighting Lab, 


2003) 


Game/Exercise Assumptions 


¢ Play will remain at the JTF level and below, 
regional powers are not and will not become 
hostile 

¢ Limited Host Nation Support, no port or 
airfield in JOA 


* Very restricted basing in theater 

¢ Timeframe: April or October? 2015 

¢ Area of Play: SE Asia 

* Classification: Unclassified 

* Force Levels: MEB / ESF equivalent 

* Game/Exercise play: Operational/Tactical 
* Sea base minimum OTH (25nm at sea) 


Red Objectives 


¢ Goals: Gain independence 
¢ Methodology: 


Survive until Central Government collapses “Fait Accompli” 
— Avoid direct conflict with Coalition forces 
— Invite NGOs into conflict area 
— Invite international media, showcase civilian deaths 
— Repeatedly stress “Caliphate” does not desire hostilities 
— Ask for UN brokered cease fire 


— Promise to act decisively to end piracy within territorial 
waters 


— Offer to hold elections in 2 years 


— Portray self as small Muslim nation attacked unjustly by rich 
Christian crusaders 


— If conflict unavoidable, attempt to draw coalition forces into a 
MOUT/Jungle fight 
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Red Order of Battle 


Air — 1ADS (mobile SA-10/20,13,8, & 67). Adding Ships with increased AD cap (HQ-61). 
AAA -— RBS-90 Bolide, 2S6M (SP 30mm/SA-19) 


TBM — CSS-2 IRBM, 6 mobile launchers, 2 x reloads, 6 decoys. 


Air Force: 72 — SU-27 Flanker, and 72 — SU-25 Frogfoot, 36 — Foxhounds, 2 B-727 AEW, 
3 KC-130 Refuelers 


Land — 6 SPF Brigades. Mobile Artillery (VTT-322, 2S23) with smart sub-munitions and 
passive counter battery capability (SORAS). PT-76 upgraded thermals, shoot on move, 
90mm. Decentralized Infantry Operations (Fire and Forget Soldiers). 


Local Constabulary and irregulars.(need to establish # & Loc) 


Naval — 6 Kilo 636 (w ASCM), patrol craft/missile patrol craft, mines, and armed militia 
in small civilian craft. 


Naval Air — 8 Folker 27’s Maritime Patrol Aircraft (Joint real-time targeting) 


Naval Coastal Defense — 3 Batteries mobile ASCM’s (YJ-82 ASCM) 1 Naval Inf Bde to 
protect ASCM and base. 


Electro-Magnetic — Jamming / spoofing capability (radio and GPS) Computer Network 
Attack? 


Space — Access to commercial space assets. Controls major Green satellite terminal 


WME - Red commercial and research facilities are capable of make biological and 
chemical weapons. 


The following force structure is for the enemy units portrayed in the Sea Viking 


04 scenario. (Marine Corps Warfighting Lab, 2003) 
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REGRESSION MODELS FOR MULTI-UAV 
SCENARIOS 


APPENDIX B. 


Anomalies discovered during data analysis prompted additional iterations of 
various design points for the multi-UAV scenarios. Due to time constraints and the need 
for still more runs, detailed analysis of the multi-UAV scenarios is not presented in this 
work. Some insights from the data may be gained, but the statistical rigor to support 
them is lacking at this point. It is highly encouraged that further research investigates this 


data. 


The model produced for the two UAV scenario, displayed below, is very similar 
to the model for the one UAV scenario, with a couple exceptions. The reactivity terms 
drop out earlier and are not included in the preferred model and a strong interaction 
between sweep width and probability of classification appears. The reactivity terms are 
statistically significant, and they could be included in the model, however they do not 
provide the level of practical significance seen in the single UAV case. The interaction 
between sweep width and probability of classification appears similarly in the decision 


tree analysis for the single UAV model. 
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The leverage plots for the two UAV scenario, displayed below, show the relative 
effects of the terms in the preferred model. Time has a positive effect here as opposed to 


the negative effect seen in the single UAV model. 


Leverage Plots for Two UAV 6 Term Model 
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The model produced for the three UAV scenario, displayed below, is similar to 
the two UAV model with the addition of a sweep width squared term and an interaction 


between time and probability of classification. 
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The leverage plots for the three UAV scenario, displayed below, show the relative 
effects of the terms in the preferred model. Sweep width is most significant as displayed 
by the steep slope of the sweep width plot. Time has a negative effect on the rate of 


enemy proportion classified, similar to the effect seen in the one UAV model. 





Leverage Plots for Three UAV 7 Term Model 
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